Implement support for replaying events#6491
Conversation
dbf91ac to
50b48ec
Compare
|
Caution Review failedAn error occurred during the review process. Please try again later. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
zoldar
changed the title
|
|
Thanks for the writeup @zoldar! Based on the description do I understand correctly that |
Correct. That is, unless the session ID is explicitly provided when running replay. This enables resuming interrupted session. |
|
Thanks! I don't fully understand why a separate salts mechanism is needed here. From what I can tell its purpose is to ensure that user_ids from live ingestion vs different replay runs do not clash. But can that not be achieved by including the replay_session_id as an input to the user_id hash function? |
|
@ukutaht That's what I basically did initially when spiking it. However, after some more research, I've concluded this is not going to cut it. In SipHash, providing a secret key with good entropy is crucial. Using |
|
So I was thinking of something like:
Is this what you had spiked as well? I understand that a randomly generated session_id is not suitable for cryptographic purposes but that's the job of the salting mechanism we already have. The salt is generated with I do see the need if you had spiked a version where the
In this version I agree the session_id is not strong enough. But then I would ask why not move the session_id to the second argument and keep the usual salt in the first. |
|
@ukutaht 🤦 Of course at first I've spiked the dumb version and somehow completely missed the possibility of altering hashed value in |
|
Ah, okay, I think that has crossed my mind actually but I've discarded the idea because salts could be cycled in the middle of replay. That's a non-issue though. I don't know, either way. That. Was. Dumb. argh |
|
@zoldar nice, thanks 👍
Hmm yeah the ingestion pipeline is designed in theory to carry sessions over during salt rotation but I'm not sure how well it works in practice tbh. |
3 participants
Changes
This PR implements support for replaying events via existing ingestion pipeline. Although the current implementation is entirely contained in
analyticsrepository, the eventual solution will be split amonganalyticsandpersistor.The plan for release is following:
replay_session_idfield and use a different cache key for replayed events as well inCacheStore; the change will be merged and released as wellImplementation details
Recording
A simplified diagram of how recording is integrated into the event ingestion pipeline:
When the event payload reach Ingress proxy, the haproxy dispatches its body and headers to the agent process listening on a tcp socket. The agent deserialises the payload, assigns a random ID to the event and asynchronously persists the event in the store while immediately returning the newly assigned ID to the proxy. The proxy appends it as
X-Replay-Event-IDheader. As of now, it's not used and is ignored down the pipeline.The analytics ingest pipeline processes the event normally without any additional processing and so does persistor, after which the event ends up batched and inserted into ClickHouse DB.
Replaying
When replaying, each payload will go through the same ingress proxy but it won't be sent to the recording agent on the basis of replay header being already present. The replay utility appends 3 following headers when sending the payloads:
X-Replay-Event-ID(ignored for now, like in the case of recording)X-Replay-Session-IDX-Replay-TimeOnce the payload reaches ingestion pipeline, it's decoded in the request building phase. There, for replayed event, the event timestamp is replaced with the one provided in
X-Replay-TimeandX-Replay-Session-IDis put underrequest.replay_session_id. The replay session ID is then carried over to event attributes.The
put_saltsstep is crucial in ensuring that replayed events are processed in complete isolation from other events even if some user sessions are still valid from the time of recording. Without that, the existing sessions will get corrupted. Processing replayed events in isolation is a trade-off we have to make - there's no feasible way to retroactively adjust existing sessions, especially somewhere in the middle.The isolation is achieved by - instead of using the salt from the database - generating a pseudo-random hash on the basis of replay session ID using
Plug.Crypto.KeyGeneratorand mixing replay session ID with secret key base in a safe way. This brings another challenge, because the hash is generated using PBKDF2 with default number of 1000 iterations, which takes around 1-2 ms when tested locally. Fetching salt from ETS cache takes only a couple microseconds in comparison. Even reducing number of iterations to 1 (which raises safety concerns over risk of exposing secret key base too easily) the timing is still strongly in favor of ETS. To work around that, we useKeyGenerator's built-in ability to cache computations using an ETS table. A separate process manages the table and purges it every 2 hours, just to be safe (though cache table blowing up is unlikely, given session IDs can't be provided externally).The last step is session management in persistor. Replayed sessions are stored and fetched using distinct shape of cache key, including replay session ID. Although risk of user ID collisions is low, it's still better to be on the safe side - that's one more level of ensuring isolation.
Lastly, when a new replayed session is provisioned, its replay session ID attribute is populated accordingly, like in the case of event.
Recording replay session ID allows relatively easy "rollback" of a botched replay and starting over again. Also, two different replays (with different session IDs), even if overlapping, are easy to tell apart and clean up properly as well. An interrupted session can also be safely resumed (the replay utility allows to provide an existing session ID; otherwise it generates a unique one on every run).
Tests